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A database system for text Tiles is employed which comprises a compiler (210) which receives as input a description of the 
source file types or languages and search filter (200) and outputs a tag set definition file and filter file (220) to be used by the 
browsing mechanism (325). The tags set definition file is input to the database builder mechanism (230) and is used by the data- 
base builder to select and translate the information from the input source file (240) to a database component file (260). Thus, by 
providing a description of the source file type, any type of source file in any language can be input to the database system of the 
present invention to generate a database component file which may be subsequently reviewed and searched by the same browsing 
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USER EXTENSIBLE, LANGUAGE SENSITIVE 
DATABASE SYSTEM 



BACKGROUND OF THE INVENTION 



1. FIELD OF THE INVENTION: 

5 The method and apparatus of the present invention relates to the 

organization of databases. More specifically, the method and apparatus of the 
present invention relates to the generation of database files from source files of 
different types. 

10 2. RELATED APPLICATION : 

This application is related to U.S. Patent Application Serial No. 

filed, - , entitled "Method and Apparatus for 

Searching Database Component Files to Retrieve Information from Modified 
15 Files" and is herein incorporated by reference. 

3. Art Background: 

A database is a collection of information which is organized and 
20 stored in a predetermined manner for subsequent search and retrieval. 
Typically, the data is organized in such a manner that the data is indexed 
according to certain parameters and can be retrieved according to those 
parameters. For example, a database may contain information to index words in 
the text file such that words or strings of words in the text file may be retrieved 
25 quickly. The data stored in the database structure may be organized in a single 
file or a multiplicity files for access and retrieval. 

Typically the database file is organized according to the type of 
source file the database is generated from. For example, a database file for a 
30 source file which is the text of a book most likely would be different from a 
database derived from source code of a computer program. Similarly, the 
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browsing mechanisms developed to search the databases would differ from 
source file type to source file type. Correspondingly, databases and the 
browsing mechanisms to search the databases vary according to the language 
of the source file. 

5 

In addition, with the constant appearance of new computing 
languages, it is extremely difficult to provide a database and browsing 
mechanism for each language. A solution to this problem is to provide a user- 
extensible database and browsing mechanism whereby a user can define the 
10 semantics of a language to the universal database and browsing mechanism. 
Once the semantics of the language are defined, a database can be built and 
searches can be performed. 
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SUMMARY OF THE INVENTION 

It is therefor© an object of the present invention to provide a database 
system which may be used for a variety of source file types and languages. 

5 

It is an object of the present invention to provide a database system 
comprising a user extensible database and browsing mechanism. 

It is further an object of the present invention to provide an interface 
10 between a source file and the database system such that the same database 
generation mechanism may be used for a variety of source file types and the 
same browsing mechanism can be used to search the database component 
files generated. 

15 A database system for text files is employed which comprises a compiler 

which receives as input a description of the source file types or languages and 
corresponding search filter and outputs a tag set definition file per language 
and a filter definition file to be used by the browsing mechanism. The tag set 
definition file is input to the database builder mechanism and is used by the 

20 database builder to select and translate the information from the input source 
file to the database component file. Thus, by providing a description of the 
source file type, any type of source file in any language can be input to the 
database system of the present invention to generate a database component 
file which may be subsequently reviewed and searched by the browsing 

25 mechanism. 
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BRIEF DESCRIPTION OF THE DRAWINGS 

The objects, features and advantages of the database system of 
the present invention will be apparent from the following description of the 
5 present invention in which: 

FIGURE 1 is a block diagram representation of an illustrative 
computer system of the present invention. 

-I o FIGURE 2 is a functional block diagram of the database system 

of the present invention. 

FIGURE 3a and 3b are functional block diagram of preferred 
embodiments of the database system of the present invention. 

15 

FIGURE 4 is an illustrative source file type description. 

FIGURE 5a and 5b are illustrative filter menus. 

20 FIGURE 6 is a flow chart of the compiler mechanism of the 

database system of the present invention. 

FIGURE 7 illustrates a filter file generated by the compiler for the 
source file type description. 

25 

FIGURE 8a and 8b illustrate a tag set definition file generated 
by the compiler for the source file type description "langl" and "lang2 M . 
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FIGURE 9a illustrates a source file to be input to the database 
system of the present invention. FIGURE 9b is an illustrative database builder 
and FIGURE 9c illustrates a database component file generated by the 
5 database system of the present invention for the source file of FIGURE 9a. 
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pETAILED DESCRIP TION OF THE INVENTION 

Notation A nd Nomenclature 

5 The detailed descriptions which follow are presented largely in terms of 

algorithms and symbolic representations of operations on data bits within a 
computer memory. These algorithmic descriptions and representations are the 
means used by those skilled in the data processing arts to most effectively 
convey the substance of their work to others skilled in the art. 

10 

An algorithm is here, and generally, conceived to be a self-consistent 
sequence of steps leading to a desired result. These steps are those requiring 
physical manipulations of physical quantities. Usually, though not necessarily, 
these quantities take the form of electrical or magnetic signals capable of being 

1 5 stored, transferred, combined, compared, and otherwise manipulated. It 

proves convenient at times, principally for reasons of common usage, to refer 
to these signals as bits, values, elements, symbols, characters, terms, 
numbers, or the like. It should be borne in mind, however, that all of these and 
similar terms are to be associated with the appropriate physical quantities and 

20 are merely convenient labels applied to these quantities. 

Further, the manipulations performed are often referred to in terms, such 
as adding or comparing, which are commonly associated with mental 
operations performed by a human operator. No such capability of a human 

25 operator is necessary, or desirable in most cases, in any of the operations 
described herein which form part of the present invention; the operations are 
machine operations. Useful machines for performing the operations of the 
present invention include general purpose digital computers or other similar 
devices. In all cases there should be borne in mind the distinction between the 

30 method of operations in operating a computer and the method of computation 

itself. The present invention relates to method steps for operating a computer in 
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processing electrical or other (e.g., mechanical, chemical) physical signals to 
generate other desired physical signals. 

The present invention also relates to apparatus for performing these 
5 operations. This apparatus may be specially constructed for the required 
purposes or it may comprise a general purpose computer as selectively 
activated or reconfigured by a computer program stored in the computer. The 
algorithms presented herein are not inherently related to a particular computer 
or other apparatus. In particular, various general purpose machines may be 
10 used with programs written in accordance with the teachings herein, or it may 
prove more convenient to construct more speciali2ed apparatus to perform the 
required method steps. The required structure for a variety of these machines 
.will appear from the description given below. 

15 General System Configuration 

Fig. 1 shows a typical computer-based system for the generation and 
searching of databases according to the present invention. Shown there is a 
computer 91 which comprises three major components. The first of these is the 

20 input/output (I/O) circuit 92 which is used to communicate information in 

appropriately structured form to and from the other parts of the computer 91 . 
Also shown as a part of computer 91 is the central processing unit (CPU) 93 
and memory 94. These latter two elements are those typically found in most 
general purpose computers and almost all special purpose computers. In fact, 

25 the several elements contained within computer 91 are intended to be 

representative of this broad category of data processors. Particular examples 
of suitable data processors to fill the role of computer 91 include machines 
manufactured by Sun Microsystems, Inc., Mountain View, California. Other 
computers having like capabilities may of course be adapted in a 

30 straightforward manner to perform the functions described below. 
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Also shown in Fig. 1 is an input device 95, shown in typical embodiment 
as a keyboard. It should be understood, however, that the input device may 
actually be a card reader, magnetic or paper tape reader, or other well-known 
input device (including, of course, another computer). A mass memory device 

5 96 is coupled to the I/O circuit 92 and provides additional storage capability for 
the computer 91. The mass memory may include other programs and the like 
and may take the form of a magnetic or paper tape reader or other well known 
device. It will be appreciated that the data retained within mass memory 96, 
may, in appropriate cases, be incorporated in standard fashion into computer 

10 91 as part of memory 94. 

In addition, a display monitor 97 is illustrated which is used to display 
messages or other communications to the user. Such a display monitor may 
take the form of any of several well-known varieties of CRT displays. A cursor 
1 5 control 98 is used to select command modes and edit the input data, and in 

general provides a more convenient means to input information into the system. 

Process D escription 

20 The user*extensible database system of the present invention includes a 

compiler and database builder. Database component files may be termed for 
different types of textual source files and the same browsing mechanism (also 
referred to as a "search mechanism") may be used for a variety of source file 
types or languages (hereinafter referred to collectively as "languages") such 
25 that the database browsing means is not limited to a particular language. For 
example, the same database can be used for a variety of computer programs 
which are in different programming languages. Similarly, database component 
files can be formed for a plurality of text materials such as books or articles 
which are written in a variety of languages as well as in a variety of styles and 
30 formats. 
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The system is illustrated by the functional block diagram of Fig. 2. A 
description of the source file language 200 (referred to as "source file type 
description") is input to a compiler 210 which generates a tag set definition 220 
which is understandable by the database builder 230. To generate a database 
5 component file for an input source file, the tag set definition 220 is input to the 
database builder 230 along with the input source file 240. The database 
builder 230 uses the tag set definition and applies it to the input source file 
when extracting the symbol information and formatting the information to be 
stored in the database component file 260. The database component file 260 
1 0 can be subsequently searched by a browsing mechanism to retrieve 
information relevant to the input source file. 

The source file type description defines the language of the source file. 
The source file type description comprises a listing of semantic tag values that 
1 5 can be associated with symbols that are to be entered into the database. The 
source file type description optionally includes filter descriptions specifically 
tailored to the type of source file which permits the user to employ various filters 
when performing searches. 

20 The semantic tags provide the means to classify portions of text of the 

source file and provide the means to use the same browsing and filtering 
mechanism for a multiplicity of source file types. The compiler uses the source 
file type description to generate a tag set definition file in a format 
understandable by the database builder 230. In addition, if filter descriptions 

25 are included in the source file type description, the compiler generates a filter 
file according to the semantic tags for the source file type which is referenced 
by the browsing mechanism to provide the user with filtering options when 
performing searches in the database for that source file language. Thus, the 
browsing mechanism is compatible with all database component files 

30 generated and modification of the browsing mechanism to search each 

particular source file language is avoided. To perform a search on a database 
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component file, the query is input to the browsing mechanism identifying the 
symbol and a filter. The filter provides a means to narrow the query and 
provide a more precise search. For further information regarding the browsing 

mechanism, see co-pending patent application U.S. Serial No. , 

5 fi| ec j t - entitled: 'Method and Apparatus for Searching Database 

Component Files to Retrieve Information From Modified Files." 

Through the use of the database system of the present invention, source 
files written in a variety of languages may be input to the database builder to 
10 generate database component files which can be subsequently searched 
utilizing the same browsing mechanism. This is illustrated by Figure 3a in 
which four input files. 300, 301, 302, 303, each written in a different language 
may be input to the database builder 315 to generate database component files 
320, 321, 322, 323 which are subsequently utilized by the browsing 
1 5 mechanism 325 to process a query 340 to generate the matches which are the 
result of the query 295. To decrease the amount of time required to perform a 
query, it is preferred that an index file 345 is generated for a database to 
provide an index of database component files each symbol is located in. The 
browsing mechanism utilizes the index file 345 to determine which database 
20 component files to search to find occurrences of a particular symbol. 



The database builder 315 in this embodiment is a single mechanism 
which generates database component files for all language types. Preferably, 
the database builder will read the source file, extract the symbols contained in 
25 the source file assign the corresponding semantic tag to the symbol according 
to a predetermined criteria (e.g. symbol type), and build a database component 
file identifying each symbol, the line number of the line where the symbol is 
located in the source file and the corresponding semantic tag. The database 
builder 315 may also be part of another mechanism which extracts the symbols 
30 from the source file whereby the database builder 31 5 utilizes the extracted 
symbol information provided to build the database component file. The 
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capability to utilize the same browsing mechanism regardless of the language 
of the source file is provided through the compiler 285 which accepts as input 
the source file descriptions for each of the source file types 270, 271, 272, 273 
and generates a tag set definition file for each source file type 280, 281 , 282, 
5 283. Optionally, filters for different source file languages can be specified in 
the source file type description and the compiler can generate a filter file . 
(referred to herein as the "ex." file) 290 which is utilized by the browsing 
mechanism 325 to enable the user to direct the browsing mechanism to perform 
filtering functions on queries. 

10 

An alternative embodiment is illustrated in Figure 3b. In this embodiment 
a separate database builder 310, 311', 312, 313 is provided for each language. 
For example, the source file "foo,1" 300 is input to the database builder 310 
along with the tag set definition file 280 to generate the corresponding 
15 database component file. Similarly, source file "foo.2" 301 is input to the 

database builder 31 1 along with the corresponding tag set definition file 281 to 
generate the database component file 321 . 

This embodiment may be used when the database generator of the 
20 present invention is incorporated into a language specific process such as a 
computer program compiler or interpreter. The computer program compiler 
analyzes the source file (i.e., the computer program to be compiled) and 
generates object code for subsequent execution of the computer program. In 
its analysis of the source file, the computer program compiler identifies each 
25 symbol and symbol type used in the computer program. The database builder 
utilizes this information to identify the corresponding semantic tags and to 
generate the database component file comprising each identified symbol, the 
line number the symbol is located on in the source file and the corresponding 
semantic tag defined in the tag set definition file. 

30 
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As briefly explained above, the source file description defines the 
semantic tags for symbols and their associated filters, all of which are source 
file type dependent. One source file description is provided for each source file 
type that is to be input through the database system of the present invention. 

5 

Referring to Fig. 4, a source file type description is described for 
theoretical languages "iangl" and "iang 2\ The file is in the ASCII format; 
however, the database system of the present invention is not limited as such 
and can be implemented in other formats. The language "Iangl" consists of two 
10 semantic tags t "first_onJine" and "not_first_dnJine" which are respectively 
defined to be a word first in a line of text and a word not first in a line of text. The 
language "Iang2" also consists of two semantic tags "capitalized_word" and 
"not„capitalized_wo^d ,, which are respectively defined to be words that are 
capitalized and words not capitalized. At the beginning of the file (line 1) is a 
1 5 "version" statement which is of the form: 

version-statement = Version' < small integer> 
The version statement takes an integer version number in the range of 1 to 
32767. When the version statement is located in the file prior to the occurrence 
of a language statement (as is the version statement at line 1), the version 
20 number indicated is used to identify the version of the .ex file containing the 
filters and the tag set definition file being produced. If the version statement is 
located in the file after a language statement (as is the version statement at line 
4), the version number is used to identify the version of the specific language. 

25 Referring to Fig. 4, line 3 contains a "Language" statement. The 

Language statement is of the form: 

language-statement:^ 'language* <name> 

30 The Language statement takes 1 argument. The argument, "name" is the 

name of the language the following statements describe. In the present 
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illustration, the name of the language is "langl". Similarly for the Language 
statement at line 24, the name of the language is "Iang2\ 

Line numbers 6 through 9 illustrate a "Properties" statement. The 
5 Properties statement declares the properties of symbols and are used to 

associate semantic tags, as set forth in the "Tags" statement (lines 13 through 
15), and filters, as set forth to in the "Filter statement (lines 17 through 20). 
Each semantic tag has a list of properties that it defines and each filter item has 
one or more lists of properties that it matches. A filter item will match all tags that 
1 0 define all properties that the filter item references. The Properties statement is 
of the form: 

property-statement := 'properties* <prefix> , {*<property-identifier>* , }\ 

15 The first argument prefix, identifies the Properties statement. The second 

argument lists the actual properties. For example, the properties of langl are 
"word", "first" and "noMirst" and the properties for Iang2 are "word", 
"capitalized" and "noncapitalized". 

20 The Tags" statement at lines 13 through 15 define a list of semantic tags 

that the database builder is supposed to associate with symbols of langl in the 
database. The Tags" statement is of the form: 

tags-statement := 'tags' <prefix> <size> , { , <tag>* '}' 
where, tag := <tag-identifier> '=* '(• <property identifier^ •)' [ <weight> ] 

25 

The argument "prefix" is used to indicate the prefix tag statement to identify the 
statement. The "size* argument identifies the number of tags and is used to 
ensure each tag statement generates the same number of tags each time the 
compiler is executed. The argument "property identifier is used to match filter 
30 items to semantic tags by associating the semantic tag with properties identified 
in the Properties statement. For example, the Tags statement for langl at line 
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13 through 15 includes the tag "first_on_line" having the properties "word" and 
"first" and the tag "not_first_onjine" having the properties "word" and 
"not_first\ The argument "weight" associates an integer value with each tag 
which is used to sort matches in response to a query. Matches having the 
5 lighter weights are presented before matches of heavier weights. 

Figure 4, lines 17-20, sets forth a Filter statement for language langl 
and lines 39-42 set forth a Filter statement for the language Iang2. The 'Filter 
statement defines a filter panel consisting of filter items and is of the form: 
10 filter-statement :=filter' '{'^Mer-item^y 
filter-item :=leaf-item | pull-right-item 

leaf-item := < " f <filter-item-name>'"'[<equiv-spec>] , = , <prop-lists> 
prop-lists :=t<property>*T[ T T<property>* T ]* 

pull-right-item := ,m <filter-item-name> , " , [<equiv-spec>] 'filter' •{* <filter-item>* '}' 
15 equiv-spec :='equiv' <language> 4m <filter-item-name>'"' 

When a filter item is built, the properties identified with the filter 
item and all semantic tags are used to figure out the semantics tags the filter 
item should match. The filter, which is specified for a particular source type, 

20 permits the user to narrow the search based on the semantic tags of the 

symbols. Preferably the filter is implemented through a menu panel displayed to 
the user such as the the one illustrated in Fig. 5a. Optionally, the fitter panel 
may be expanded to include additional filter items which can be used to narrow 
a search. For example, Fig. 5b illustrates a filter panel expanded after selection 

25 of the filter item "Declaration" 

The filter panel consists of a list of filter items. Each filter item defines a 
list of semantic tags that the filter should match. This is built from the property 
list ("prop-lists") of the filter items and the semantic tags. Each filter item is 
30 either a "leaf" item or a "pull-right" item. The leaf filter item specifies one or 

more properties that are used to identify the semantics tags the filter item should 
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match in order to be considered to be matches when the browsing mechanism 
performs a search. 

Pull-right fitter items specify another fitter statement which is displayed 
5 when a particular filter item defining a pull-right menu item is selected and 
inherit the set of tags to match from the first leaf item of its pull-right filter. 

The association between tags and filter items is established according to 
the properties of a symbol. Each semantic tag has a list of properties that it 
1 0 defines and each filter items has one or more list of properties that it matches. A 
filter item will match all semantic tags that define all properties that the filter item 
references. 

Thus, when a user selects a filter item from the filter items provided in the 
15 browsing mechanism through the .ex file, queries will be restricted to symbols 
with one of the tags the filter item specifies. 



The Filter statement set forth at lines 17-20 defines three filter items "All 
20 Matches", "First" and "Not First". The first filter item is defined by the property 
"word". The second filter item "First", is defined by the properties "word" and 
"first". The third filter item "Not First" is defined by the properties "word" and 
"not_first". Thus, for example, if a search for is performed in a source input file 
written in the language langl and the filter item "First" is selected, the browsing 
25 mechanism will fitter out all symbols except those identified by the semantic tag 
first_on_Iine. 

The Filter statement at lines 39-42 define the filter items for the language 
Iang2 at line 40 r the filter item "All Words" is defined by the property "word* 1 . In 
30 addition, the filter item "all words" is also set to be equivalent to the filter item 
"All Matches" in langl. 
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The "equiv" statement permits filters to be shared among different 
source file types. The equiv statement is of the form: 

equiv-spec := , equiv t <language> ' " '^ilter-item-name^ " ' 
5 The statement is used to indicate that the current filter item is equivalent to the 
specified filter item. Thus, to display the filter, the browsing mechanism simply 
utilizes the filter corresponding to the filter item specified. When filter items 
have been equivalenced, the browsing mechanism is required to keep track of 
the set of matching semantic tags for each language. To do this, the .ex file, 
10 the file generated by the compiler, contains a vector <Ianguage-id, filter item 
name, tag-vector> triples for each filter item. The browsing mechanism is then 
able to use the correct tag filter for the language of the source file because the 
database component file contains the name of the language it was generated 
from. 

15 

Thus, the filter item "All Words" utilizes the filter for "All Matches" in 
langt. The browsing mechanism, when performing a query, for "All Matches" 
or "All Words" will also perform a query with respect to the other filter item. 

20 Referring to Fig. 6, the process flow for the compiler is explained. From 

the source file description for the particular source file type, at block 500 the 
compiler collects general information regarding the source type description, 
such as the version number. 

25 At block 510, the Language statement is collected from the source file 

type description to identify the name of the language for the source file type 
description. Each statement, at block 520, is examined to determine the type of 
statement e.g. Properties statement, Tag statement or Filter statement. If the 
statement type is a Properties statement, at block 530, the corresponding 

30 properties information in the statement is collected from the source file type 
description. If the statement type is a Tag statement, at block 540, the tag 
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information is collected. Similarly, if the statement is a Filter statement, at block 
550, the filter information is collected. 

After collecting properties information, tags information and filter 
5 information, at block 560, if another language statement is found, the process 
continues for another source file type (more than one source file type 
description may be contained in a single file, as is illustrated by Fig. 4). If the 
next statement is not a language statement, the statement must be a Properties, 
"Tag or Filter statement whereby processing continues at block 520 via block 
1 0 565. If, at block 565, the end of the file is reached, the fitters defined by the 
Filter statement(s) are built, block 570, and at block 580, are written into the 
Filter file (.ex file). At block 590, one tag set definition file is written per source 
file type description. 

15 The .ex file generated for language langl defined by the source file type 

description of Fig. 4, is illustrated by Fig. 7. At the beginning of the .ex file is a 
header section 700 containing a listing and location of the sections forming the 
file. The language section 710 identifies the languages defined. The weight 
section, 715, 720 specifies the weights of the semantic tags for such language. 

20 Preferably, as is the case in the present illustration, the weights are identified in 
the tags section 740 and 750. The filter section 730, specifies the six filter items 
"All Matches", "All Words", "First", "Not First", "Capitalized" and "Not 
Capitalized". Each filter item identifies the language the filter applies to, the 
name of the filter item, and the semantic tags that match the filter item. The filter 

25 items, "All Matches" and "All Words" were specified to be equivalent. This is 
reflected in the .ex file at section 735 where the two filter items are specified to 
be the same item on the filter menu. Although not illustrated, if pull-right filter 
items were specified in the source file type description, the filter section would 
reflect this by setting the "Pullright Offset" parameter of the filter item having a 

30 pullright filter to a value which identifies an offset in the file where the filter 

items, which are available when the pullright filter item is selected. a*e located. 
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The tags section 740 and 750 specifies each semantic tag and its 
weight. This section is optional in the .ex file because the semantic tags are 
identified in the tag set definition and is provided in the .ex file for use during 
5 debugging. Each semantic tag is identified by its name, value (also referred to 
as semantic tag number) and weight. The final section, the release section 760 
identifies the release version for the browsing mechanism. 

The tag set definition identifies each semantic tag for the language. The 
10 resulting tag set definition files generated for the languages langl and Iang2 

from the source file type description of Figure 4 are illustrated in Figures 8a and 
8b. Each file includes an "enum" declaration (an enum declaration is used to 
declare constants in the C programming language) that lists all semantic tags 
for the language that can be associated with symbols. Although the tag set 
1 5 definition is illustrated here as a file written in the C programming language, the 
tag set definition file is not limited as such and can be implemented in a variety 
of formats including a simple list of semantic tags. 

Specifically, with reference to Fig. 8a, the enum declaration identifies 
20 the semantic tags for langl, "first-on-line" and "not_first_onJine". The 
semantic tags are specified according to : <prefix>"_"<language 
name>"_"<semantic tag name>"="<semantic tag number>. It should be noted 
that two dummy tags "langljill" and "Iang1_fill" are inserted into the enum 
declaration to bring the total number of tags to four, the maximum number of tags 
25 specified by the Tags statement in the Source File type Description (Fig. 4). In 
addition, the declaration contains two additional tags, 
cbjang1_tags_first_member and cb_lang1_1ast_member, used to delineate 
the beginning and end of the declaration. 

30 Similarly, with respect to Fig. 8b, the enum declaration identifies the 

semantic tags for Iang2, "capitalized_word" and "not_capitalized_w3rd" ss 
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well as two dummy tags to bring the number of semantic tags to the maximum 
tag size. 



The tag set definition files generated by the compiler are input to the 
5 database builder to generate the database component file for an input source 
file. Figure 9a illustrates a sample input source file written in the language 
langl which is simply English text having two semantic tags 
"cbJang1_first_onJine" and H cbJang1_not_first_pnJine\ Figure 9b is an 
illustrative database builder for the language langl written in the C 
10 programming language. The database builder examines each symbol (in the 
present illustration, each word is a symbol) contained in the input source file 
and determines whether the symbol is the first symbol on a line of text. If it is first 
on a line of text, the database builder assigns to the symbol the semantic tag 
cb_lang1_first_pnjine and puts this information in a database component file 
1 5 for subsequent access by the browsing mechanism for performing queries. 

Similarly, if the symbol is not the first symbol on a line of text in the source input 
file, the database builder assigns to the symbol the semantic tag 
cb_Jang1__not_first_pnJine and this information is placed in the database 
component file. 

20 

Figure 9c illustrates the database component file generated for the 
source input file of Figure 9a. The database component file contains a header 
section 800, source name section 810, symbol table section 820, semantic 
table section 830 and release section 840. 

25 

The header section 800 contains administrative information such as the 
magic number of the file (common to UNIX files) and file version numbers. The 
field case should be set in database component files that were generated by 
database builders that ignore the case of characters. The language field 
30 contains a string that is used to select the correct filter item when querying, in 
the present example, language langl. The source type field indicates if the file 
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is a top level source file or an included file. The rest of the header is a table of 
contents for the file where the type, size and location of all the sections are 
listed. 

5 The source name section 810 contains the absolute path to the source 

file the database component file was built from, tt also contains a bit that is set 
to a value of one if the path for the file was relative. This is used by the source 
browser when rendering file names. 

I o The symbol table section 820 is a randomly ordered list of all the 

symbols described in this database component file. Each symbol carries a file 
relative pointer corresponding to the vector of semantic records set forth in the 
semantic table section 830. 

15 The semantic table section 830 contains a vector of <Iine-number and 

semantic tag> pairs for each symbol listed in the symbol table section. Both the 
numeric and string value of each tag is displayed. The vectors appear in the 
same order as the symbols in the symbol table. Each vector is terminated by an 
end of vector record. "End Record". For example, the vector identified at line 
20 1060 for the syrhbol "ourselves" is assigned the semantic tag 

cbjangl Jirsronjine because, as shown in Fig. 9a, the symbol "ourselves" 
is the first word on line 4 in the source input file (Figure 9a). The vector listed in 
Figure 9c at line number 1084, is the vector for the symbol "the". In all but one 
occurrence, the symbol identified at line 1104, which occurs at line 5 in the 
25 source input file, the symbol "the" is identified as having the semantic tag 
cb_lang 1 _not_f i rst_o n Jine . 

The final section, the release section 840, identifies the database system 
release version and date. 

30 
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While the invention has been described in conjunction with the preferred 
embodiment, it is evident that numerous alternatives, modifications, variations 
and uses will be apparent to those skilled in the art in light of the foregoing 
description. 
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ClMMS 

What is claimed is: 

5 1 . In a computer system comprising a CPU, input/output means and 

memory containing a file system, said file system comprising at least one 
source file comprising text, a user-extensible database system in which source 
files of different source file types can be input into the database system and 
queries can be performed on the the source files input using the same browsing 

10 mechanism, said database system comprising: 

source file type definition which defines the language of the source file 
to be input to the database system, said source file type definition comprising; 

a properties statement which identifies properties of the language; 
15 a tags statement which identifies semantic tags of the language, 

said semantic tags defined according to at least one property; 

a filter statement which identifies at least one filter item for the 
languagfe, said filter item defined according to at least one property 
compiler means for compiling the source file type definition to generate a 
20 filter description and tag set definition; 

said tag set definition comprising a listing of each semantic tag for 
the language; 

said filter description comprising a tisting of the filter items for a 
language, each fitter item comprising a filter name, the language in 
25 which the filter is used and at least one semantic tag determined 

according to the properties specified by the tags statement and filter 
statement in the source file definition; 

database builder means to generate a database component file for a 
source file of a particular source file type, said database builder receiving as 
30 input the source file and the tag set definition for the source file type, said 
database builder comprising: 
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means for identifying each symbol in the source input file; 
means for identifying the semantic tag, as set forth in the tag set 
definition file, for each symbol identified; 

means for storing in the database component file the symbol, the 
5 line number where the symbol occurs in the source file and the semantic, 

tag for each symbol occurrence in the source input file; 
database browsing means for searching the database comprising at 
least one database component file in response to a query to find a symbol, said 
query identifying the symbol to search for and a filter item, said database 
10 browsing means comprising; 

means for searching a database component file for each 
occurrence of the symbol identified by the query; 

means for comparing the semantic tags of each occurrence of the 
symbol to the semantic tags specified for the filter item identified by the 
15 query; 

means for generating as a result of the query those occurrences 
of the symbol identified by the query which has a semantic tag which 
matches one of the semantic tags of the filter item identified by the query; 
whereby different types of source files can be input into the database 
20 system by providing a source file type definition and the same database 

browsing means is used to perform queries regardless of the input source file 
type. 

25 2. The database system as set forth in claim 1 , further comprising 

multiple database builder means, one database builder means for each 
language of source files 

3. The database, system as set forth in claim 2, wherein a source file 
30 type definition defines a computer programming language and the database 
builder means is part of a computer programming language compiler. 
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4. The database system as set forth in claim 1 , said means for 
searching a database component file further comprising an index file 
comprising a listing of each symbol and the database component file the 

5 symbol is located in, said means using the index file for searching only those 
database component files containing the symbol identified by the query. 

5. The database system as set forth in claim 1 , said filter description 
further comprising a menu generation means to generate a menu displaying the 

1 0 filter items whereby the filter items can be selected to be the filter for a query. 

6. The database system as set forth in claim 1 wherein said filter 
description comprises a multiplicity of filter items and more than one filter item 
can be selected to determine the result of a query. 

15 

7. In a computer system comprising a CPU, input/output means and 
memory containing a file system, said file system comprising at least one 
source file comprising text, a user-extensible process for generating a 
database from source files of different source file types and performing queries 

20 on the source files input using the same browsing mechanism, said process 
comprising the steps of: 

creating a source file type definition which defines the language of the 
source file to be input to the database system, comprising the steps of; 
25 generating a properties statement which identifies properties of 

the language; 

generating a tags statement which identifies semantic tags of the 
language, said semantic tags defined according to at least one property; 
generating a filter statement which identifies at least one filter item 
30 for the language, said filter item defined according to at least one 

property; 
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compiling the source file type definition to generate a filter description 
and tag set definition; 

said tag set definition comprising a listing of each semantic tag for 
the language; 

5 said filter description comprising a listing of the filter items for a 

language, each fitter item comprising a filter name, the language in 
which the filter is used and at least one semantic tag determined 
according to the properties specified by the tags statement and filter 
statement in the source file definition; 
10 building a database component file for a source file of a particular 

source file type, comprising the steps of: 

receiving as input the source file and the tag set definition for the 
source file type, 

identifying each symbol in the source input file; 
15 identifying the semantic tag for each symbol, said semantic tag 

being one of those defined in the tag set definition file; 

storing in the database component file the symbol, the line number 
where the symbol occurs in the source file and the semantic tag for each 
symbol occurrence in the source input file; 
20 searching the database comprising at least one database component file 

in response to a query to find a symbol, said query identifying the symbol to 
search for and a filter item, comprising the steps of; 

searching a database component file for each occurrence of the 
symbol identified by the query; 
25 comparing the semantic tag of each occurrence of the symbol to 

the semantic tags specified for the filter item identified by the query; 

generating as a result of the query those occurrences of the 
symbol identified by the query which have semantic tags which match 
the semantic tags of the filter item identified by the query; 
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whereby different types of source files can be input by providing a 
source file type definition and queries can be performed regardless of the input 
source file type. 

5 
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version 1 

language langl 
version 2 

properties langl { 
word 
first 

not first 



tags langl 4 { 

first_on_line = (word first) 
not_first_on_line = (word no 

fitter { 

"All Matches" = (word) 

"First" = (word first) 

"Not First" = (word not_first) 



language Iang2 
version 3 

properties Iang2 { 
word 

capitalized 
not_capitalized 



tags Iang2 4 { 

capitalized_word = (word capitalized) 
not_capitalized_word = (word noncapitalized) 

filter { 

"All Words" equiv langl "All Matches" = (word) 

"Capitalized" = (word capitalized) 

"Not Capitalized" = (word noncapitalized) 



Fig. 4 
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Dump of sun__source_browse ex 
++++ Section ('Header' language* All languages' start=0 length«2641 
Magic number « 0x3c65783e *<ex>' 
Version number =11 
Section count * 8 

Section #0 ('Header language=«AH languages' start«0 tength«264) 

l!2!nn II SiSSSS^**" ,an fl ua ? e «' A « languages' start-264 length-44) - 700' 

Section #2 ( Weight' Ianguage«'lang1 ' start-308 length-8) 
Section #3 ('Weight' Ianguage«'lang2' start-316 length-8) 
If 2 ! Sn ti S5 iner ^ en u' language-'AII languages' start-324 tength-230) 
Section #5 (Tags' language-'langl ' start-556 length-95) ' 
Section #6 (Tags' Ianguage«'lang2' start-652 tength-101) 
Section #7 ( Release section' language-" start«756 length-50) 
O^^an^^erei O it'^ 0 2 a0eS, tan9ua 9 em ' A " pages' start-264 length-44) 
1 : *lang2' version-1^3 * 710 

++++ Section ('Weight' language='lang 1 ' start-308 length-8) = 
Reported as part of the tags section 715 

Section ('Weight' Ianguage='lang2' start=31 6 length-8) -J " 

Reported as part of the tags section ~~| 720 

5JJ.\^ i RHer/Mrn^ 1enU ' ,an9ua 9 es,AI1 'anguages' start-324 length-230) . 
328: Pullright offset - 0 
lle^n^' '£!! wSSSilrS lan 9l_nrst on linen langl_not_first_on line:l ] -» 
PiiPul^htof^etto I ,an 9 2 - ca P't a "^d.word:1 lanfl2_no U capitili 2 ed^ord:lj}. -735 
384: 'langl'Tirst'-'Ilangl first on line:1 ] 

404: Pullright offset « 0 ~ " ~ 730 

V£' « ,. ,,an9r ' Not Rrst ' " I 'ang1_not first on line:1 1 
432: Pullright offset -0 ~ ~ "~ J 

« H ,a 1? 2 ' 'Capitalized* " [ Iang2_capitalized word:1 ] 
460: Pullright offset - 0 ~~ 

468: •Iang2' 'Not Capitalized' " ( lang2_not_capitalized word ! 1 
Section (Tags' language-'langl ' stan>556 length*95) 

1 : cbJang1_first_on_Jine weight=64 
2: cbJang1_not_first_on_line weight=64 - 7an 

3: cbjang 1_lang1_fill_3 weight=64 
4: cbjangl Jangl_fill__4 weight=64 

++++ Section (Tags' Ianguage='lang2' start=652 length-1 01 ) 

1 : cbJang2_capitalized_word weight=64 
2: cbJang2_not_capitalized_word weight=64 . 7cn 

3: cbjang2_lang2_fill_3 weight-84 
4: cbjang2_lang2_fill_4 weight=64 

++++ Section ('Release section' language*" start=756 lenath-50) 

Release '@(#)RELEASE SC1 .0 SourciBrowser 1 .1 Talpha 9o7o3/02' - 760 

Fig. 7 
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enum Cbjang1_tags_tag { 

cbjang1_tags_first_member = 0, 



cbJangl_first_on_line = 1, 
cbJang1^noMirst_onJine « 2, 
cbjangl Jang1_filL3 = 3, 
cbjangl Jang1_filL4 = 4, 

cbjang1_tagsjast_nriember = 5 



enum Cbjang2_tags_tag { 

cbjang2_tags_first_member = 0, 



cbJang2_capitaiized_word = 1 , 
cbJang2_not_capitalized_word = 2, 
cbjang2jang2_fi]|_3 = 3, 
cbJang2_lang2_filL4 = 4, 

cbjang2_;tagsjast_mernber = 5 



Fig. 8a 



Fig. 8b 



SUBSTITUTE SHEET 



WO 91/14991 





PCT/US91/01673 .. 



10 / H 



We, the people of the United States, in order to form a more perfect Union, 
establish justice, insure domestic tranquility, provide for the common 
defense, promote the general welfare, and secure the blessings of liberty to 
ourselves and our posterity, do ordain and establish this Constitution for 
the United States of America. 



FIG. 9 a 
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r 

* Short program to show how a simple collector works 
7 

♦include <stdio.h> 
♦include <ctype.h> 
♦include "cbjang1_tags.h w 
♦include "cb^enter.h" 

char *cb_program_name = "langl 



void 

main(argc, argv) 
Int argc; 
char *argvQ; 

{ ' 

FILE *in; 
char word[1000]; 
char *p = word; 
int chr; 
int doing_word; 
int lineno = 1 ; 

Cbjang1_tags tag = cbJang1_first_onJine; 

if ((argc != 2) || ((in » fopen(argv[1], V)) « NULL)) { 
^ exit(1 ); /* Could not open infiie 7 

cb_enter_start(&lineno. r Init collector support package 7 

CB_CURRENTJ_ANGUAGE, 

CB_CURRENTJ^AJOR_yERSION, 

CB_CURRENT_MINOR_VERSION); 
cb_enter_pushjile(argv[1], 1); r Start collecting for file 7 

while ((chr = getc(in)) != EOF) { 
if (isspace(chr)) { 
if (doing_word) { 
•p-0; 

r 

• Collect one symbol 
7 

cb_enter_symbol(word, lineno, tag); 
p = word; 

tag = cb_lang1_not_first_onJine; 
doing_word = 0; 

} 

if (chr==^ f ){ 

tag = cbJang1_first_onJine; 
Iineno++; 

} 

continue; 

} 

doing_word = 1 ; 
•p++ = chr; 

} 

cb_enter_pop_file(); /* End collection and create database file 7 



Fig. 9b 
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-h-m< Header section (id=1, start-0,!ength-136) 
Magic number = 0x3c63623e *<cb>' 
ixJ format version number - 2,1 
#1ine seen in source - 0 
case was folded - 0 
Language: 'langV - version 1,2 
Source type - root « 1 
Number of sections « 5 
Header section (id«1, start-O, length-136) 
Source name section (id-2, start-136. Iength*48) 
Symbol table section (id=4, start- 184, length=B12) 
Semantic table section (id-5, start-996. length-412) 
Release section (id»7, start-1408. Iength=50) 
-M-++ Source name section (id=2, start-136, length-48) 
relative-1 /home/jacomo/soren/sb/patent/example.langl 
+++♦ Symbol table section (id=4, start-184, length-812) 
184: *N #a95/example.langV offset « 996 
216: 'do' offset - 1004 
228: •for* offset - 1012 
240: 'ordain' offset « 1 024 
256: 'to' offset- 1032 

268: 'Al example.lango #a95/example.langr offset - 1044 

312: 'domestic' offset = 1052 

328: 'ourselves' offset = 1060 

348: 'a' offset = 1068 

360: 'liberty' offset = 1 076 

376: 'the* offset = 1084 

388: 'P #a95/example.langr offset - 1112 

420: 'more' offset - 1120 

432: 'form' offset - 1 128 

444: 'provide* offset - 1 136 

460: 'States' offset - 1 144 

476: 'tranquility,' offset = 1 152 

496: 'promote' offset - 1 1 60 

512: 'F S /home/jacomo/soren/sb/patent/examp!e.lang1 0 .' offset 

572: 'States,' offset = 1 176 

588: 'F G langl 0 .* offset - 1 184 

612: 'Union,' offset - 1 192 

628: 'this* offset = 1200 

640: 'people' offset « 1208 

656: 'posterity.' offset = 1216 

676: 'secure' offset = 1 224 

692: 'United* offset - 1232 

708: *N example.lango' offset - 1244 

732: 'insure' offset = 1252 

748: 'general' offset - 1260 

764: 'America.' offset = 1268 

780: 'perfect* offset = 1276 

796: 'common' offset - 1284 

812: 'defense.' offset = 1292 

828: 'justice,' offset - 1300 

844 : 'welfare.' offset - 1 308 

860: 'We.' offset - 1316 

872: 'establish' offset = 1324 

892: 'our' offset = 1336 
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904; 'order' offset- 1344 

920: 'and 1 offset « 1352 

932: 'in 1 offset « 1368 

944: 'of offsets 1376 

956: •Constitution 1 offset ~ 1392 

976: •blessings' offset «= 1400 

* m a Semantic table section (id«5, start«996 t length-412* 
996: Symbol on line 1: 'N #a95/example.langr <Unknown tag> 
1000: End record * 
1004: Symbol on line 4: 'do* cbjangl notJirst_onJine - 2 
1008: End record " 
1012: Symbol on line 2 
1016: Symbol on line 4 
1020: End record 
1024: Symbol on line 4 
1028: End record 
1032: Symbol on line 1 
1036: Symbol on line 3 
1040: End record 
1044: Symbol on line 1 
1048: End record 
1052: Symbol on line 2 
1056: End record 
1060: Symbol on line 4 
1064: End record 
1068: Symbol on line 1 
1072: End record 
1076: Symbol on line 3 
1080: End record 
1084: Symbol on line 1 
1088: Symbol on line 1 
1092: Symbol on line 2 
1096: Symbol on line 3 
1100: Symbol on line 3 
1104: Symbol on line 5 
1108: End record 
1112: Symbol on line 1 
1116: End record 
1120: Symbol on line 1 
1124: End record 
1128: Symbol on line 1 
1132: End record 
1136: Symbol on line 2 
1140: End record 
1144: Symbol on line 5 
1148: End record 
1152: Symbol on line 2 
1156: End record 
1160: Symbol on line 3 
1164: End record 
1168: Symbol on line 1 



72 



•for* c*Jang1_not_first_onJlne * 2 
tof cbjangl jnotjfirstjonjine » 2 

'ordain 1 cfcJang1_notJirst_onJlne « 2 

to' cbjangl _notJirst_onJIne « 2 
'to' cbJang1_notJirst_onJine ~ 2 

'Al example Jango #a95/example.langr <Unknown tag> 

'domestic 1 c*>jang1_notjirst_onjine « 2 

'ourselves' cbjang 1 Jirst_onJlne - 1 

'a' cbJangl_notJlrst_onJine - 2 

'liberty* cbjang1_notjirst_onjlne - 2 

'the' cbJang1_notJirst_onJIne « 2 
'the' cbJang1_notJirst_onJine « 2 
'the 1 cbjangl _notJirst onjine « 2 
'the' cbJang1_notJirsfpnJine - 2 
'the' cbJang1_notJirst_onJine • 2 
'the* cbjangl Jirst.onjine = 1 

'P #a95/example.langr <Unknown tag> - 73 

'more' cbjang1_notjirst_onjine « 2 

'form 1 cbjangl _notJirsLonJine « 2 

'provide' cfcjangl j>otJirst_pnJine « 2 

'States' cbjangl jiotjirstjonjine « 2 

'tranquility,' cbJang1_notJirst_onJine « 2 

'promote' cbJang1_notJirst_onJine « 2 

'FS/home/jacomo/soren/sb/patent/example.langl 0 / 
*23 



* 830 



< Unknown tag> 
1172: End record 
1176: Symbol on line 1: 'States,' cbJang1_notJirst_onJine = 
1180: End record 

1184: Symbol on line 1 : T G langl 0 / <Unknown tag> = 30 
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11 88: End record U / U 

11 92: Symbol on line 1: 'Union.' cbjangl not first on line* 2 
1196: End record - - _ _ 

1200: Symbol on line 4: this' cbJang1_not first on fine - 2 
1204: End record - - - 

1208: Symbol on line 1:'peop!e' cbjangl not first on fine-* 2 
1212: End record - - _ 

]™ : f^ bo,on,ine4:, P oster,t y'' cb - ,an 8l_not first on line* 2 
1220: End record ~ ~ ~ . 

1224: Symbol on line 3: 'secure* cbjangl not first on line* 2 
1228: End record - - _ _ 

1232: Symbol on line 1 : 'United' cbjangl _notJirst on line « 2 
1236: Symbol on line 5: 'United' cbjangl not firsf online - 2 
1240: End record - - _ _ 

1244: Symbol on line 1 : 'N example.lango' <Unknown tao> * 71 
1248: End record 

1252: Symbol on line 2: 'insure' cbjangl not first on line * 2 
1256: End record - - - _ 

1260: Symbol on line 3: 'general' cb langl not first on line* 2 
1264: End record ~ ~ ~ ~ 

1268: Symbol on line 5: 'America.' cbJang1_not first on line * 2 
1272: End record — — _ 

1276: Symbol on line 1 : 'perfect' cbjangl not first on line « 2 
1280: End record - - _ 

1284: Symbol on line 2: 'common' cb Jang 1_not first on line* 2 
128B: End record - — _ 

1292: Symbol on line 3: 'defense,' cbjangl Jirst on line * 1 
1296: End record ~ ~ 

1300: Symbol on line 2: 'justice.' cbJang1_not first on line « 2 
1304: End record - — — 

]^! : f ym ^ lonline3:,we,fare «'^- lan 0 1 - n ot tost on fine - 2 
1312: End record - _ _ 

1316: Symbol on line 1 : "We," cbjangl first on line - 1 
1320: End record - - _ 

1324: Symbol on line 2: 'establish' cbjangl Jirst_on line * 1 
3i£S : fy m bol on line 4: 'establish' cbJang1_notJirst_ on line - 2 
1332: End record ~~ 

f ymbo1 on ,ine 4: ' our ' cbjangl not first on line - 2 
1340: End record ~ ~ ~ 

J2?» : f ymboi on ,ine 1 : ' order ' cbJang1_not_first on line - 2 
1348: End record — ~ 

221 f ymbo1 on line 3: ' and ' cbJanOl-notJIrsLonJIne ■ 2 
1356: Symbol on line 4: 'and' cbJang1_notJirst_onJine ■ 2 
1360: Symbol on line 4: 'and' cbjangl not first on line - 2 
1364: End record ~ ~ ~ ~ 

1368: Symbol on line 1 : 'in* cbJang1_not first on line « 2 
1372: End record — - - 

1376: Symbol on line 1 : 'of cbJang1_notJlrst_onJlne - 2 
1380: Symbol on line 3: 'of cbjang1_notjirst_onjine * 2 
1384: Symbol on line 5: 'of cbjangl not first on line * 2 
1388: End record ~ . ~ ~ ~ 

? ymbo1 on ,ine 4: 'Constitution' cbJang1_not first on line * 2 
1396: End record ~ ~ 

1400: Symbol on line 3: 'blessings' cbJang1_not first on line * 2 
1404: End record _ _ _ 

++++ Release section (id=7, start=1408. Iength=50) 
Release '@(#)RELEASE Sd.OSourceBrowser 1.1 alpha 90/03/02' 
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